Language tasks involving character-level manipulations (e.g., spelling correction, many word games) are challenging for models based in subword tokenization. To address this, we adapt the interchange intervention training method of Geiger et al. (2021) to operate on type-level variables over characters. This allows us to encode robust, position-independent character-level information in the internal representations of subword-based models. We additionally introduce a suite of character-level tasks that systematically vary in their dependence on meaning and sequence-level context. While simple character-level tokenization approaches still perform best on purely form-based tasks like string reversal, our method is superior for more complex tasks that blend form, meaning, and context, such as spelling correction in context and word search games. Our approach also leads to subword-based models with human-intepretable internal representations of characters.
translated by 谷歌翻译
Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning. Yet, they fail miserably on the recently proposed Winoground dataset, which challenges models to match paired images and English captions, with items constructed to overlap lexically but differ in meaning (e.g., "there is a mug in some grass" vs. "there is some grass in a mug"). By annotating the dataset using new fine-grained tags, we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images. In this paper, we identify the dataset's main challenges through a suite of experiments on related tasks (probing task, image retrieval task), data augmentation, and manual inspection of the dataset. Our analysis suggests that a main challenge in visuolinguistic models may lie in fusing visual and textual representations, rather than in compositional language understanding. We release our annotation and code at https://github.com/ajd12342/why-winoground-hard .
translated by 谷歌翻译
开发对手挑战NLP系统的方法是提高模型性能和解释性的有前途的途径。在这里，我们描述了团队在第一个动态对抗数据收集（DADC）的任务1中“长角牛”的方法，该研讨会要求团队手动欺骗一个模型，以挖掘出挖掘的问题回答任务。我们的团队首先结束，模型错误率为62％。我们主张采用系统的，语言知情的方法来制定对抗性问题，并描述了试点实验的结果以及我们的官方提交。
translated by 谷歌翻译
语法提示有时具有自然语言的单词含义。例如，英语单词顺序规则限制了句子的单词顺序，例如“狗咀嚼骨头”，即使可以从世界知识和合理性中推断出“狗”作为代理人和“骨头”的状态。量化这种冗余的发生频率，以及冗余水平如何在类型上多样化的语言中变化，可以阐明语法的功能和演变。为此，我们在英语和俄语中进行了一个行为实验，并进行了跨语言计算分析，以测量从自然主义文本中提取的及物子句中语法线索的冗余性。从自然发生的句子中提取的主题，动词和物体（按随机顺序和形态标记）提出了英语和俄罗斯说话者（n = 484），并被要求确定哪个名词是该动作的推动者。两种语言的准确性都很高（英语约为89％，俄语为87％）。接下来，我们在类似的任务上训练了神经网络机分类器：预测主题对象三合会中的哪个名义是主题。在来自八个语言家庭的30种语言中，性能始终很高：中位准确性为87％，与人类实验中观察到的准确性相当。结论是，语法提示（例如单词顺序）对于仅在10-15％的自然句子中传达了代理和耐心是必要的。然而，他们可以（a）提供重要的冗余来源，（b）对于传达无法从单词中推断出的预期含义至关重要，包括对人类互动的描述，在这些含义中，角色通常是可逆的（例如，雷（Ray）帮助lu/ Lu帮助雷），表达了非典型的含义（例如，“骨头咀嚼狗”。）。
translated by 谷歌翻译
英国主导填字游戏，英国的主要填字游戏是推进寻求语义复杂，高度成分语言的NLP系统的有希望的目标。密码线索像流利的自然语言一样读，但是由两部分进行前列组成：一个定义和播放密码需要字符级操作。专家人类使用创造性的智能来解决隐秘，灵活地结合语言，世界和领域知识。在本文中，我们进行了两个主要贡献。首先，我们向NLP系统提供了一个充满挑战的新基准，以便在更具创造性，人类的方式处理组成语言的挑战新的基准。在显示三种非神经方法和T5之后，一种最先进的神经语言模型，无法实现良好的性能，我们提出了第二种主要贡献：一种新的课程方法，其中模型首先进行微调在相关任务，如解读单词。我们还介绍了一个具有挑战性的数据拆分，检查子字标记模型的元语言功能，并通过扰动线索的晶片部分来调查模型系统性，表明T5表现出与人类解决部分符合的行为策略。虽然我们的课程方法大大提高了T5基线，但我们最好的模型仍然无法概括为人类可以的程度。因此，隐秘填字游戏仍然是NLP系统的未解决挑战和未来创新的潜在来源。
translated by 谷歌翻译
While the capabilities of autonomous systems have been steadily improving in recent years, these systems still struggle to rapidly explore previously unknown environments without the aid of GPS-assisted navigation. The DARPA Subterranean (SubT) Challenge aimed to fast track the development of autonomous exploration systems by evaluating their performance in real-world underground search-and-rescue scenarios. Subterranean environments present a plethora of challenges for robotic systems, such as limited communications, complex topology, visually-degraded sensing, and harsh terrain. The presented solution enables long-term autonomy with minimal human supervision by combining a powerful and independent single-agent autonomy stack, with higher level mission management operating over a flexible mesh network. The autonomy suite deployed on quadruped and wheeled robots was fully independent, freeing the human supervision to loosely supervise the mission and make high-impact strategic decisions. We also discuss lessons learned from fielding our system at the SubT Final Event, relating to vehicle versatility, system adaptability, and re-configurable communications.
translated by 谷歌翻译
Language models have become increasingly popular in recent years for tasks like information retrieval. As use-cases become oriented toward specific domains, fine-tuning becomes default for standard performance. To fine-tune these models for specific tasks and datasets, it is necessary to carefully tune the model's hyperparameters and training techniques. In this paper, we present an in-depth analysis of the performance of four transformer-based language models on the task of biomedical information retrieval. The models we consider are DeepMind's RETRO (7B parameters), GPT-J (6B parameters), GPT-3 (175B parameters), and BLOOM (176B parameters). We compare their performance on the basis of relevance, accuracy, and interpretability, using a large corpus of 480000 research papers on protein structure/function prediction as our dataset. Our findings suggest that smaller models, with <10B parameters and fine-tuned on domain-specific datasets, tend to outperform larger language models on highly specific questions in terms of accuracy, relevancy, and interpretability by a significant margin (+50% on average). However, larger models do provide generally better results on broader prompts.
translated by 谷歌翻译
Recent methods demonstrate that data augmentation using counterfactual knowledge can teach models the causal structure of a task, leading to robust and generalizable models. However, such counterfactual data often has a limited scale and diversity if crowdsourced and is computationally expensive to extend to new perturbation types if generated using supervised methods. To address this, we introduce a new framework called DISCO for automatically generating high-quality counterfactual data at scale. DISCO engineers prompts to generate phrasal perturbations with a large general language model. Then, a task-specific teacher model filters the generation to distill high-quality counterfactual data. We show that learning with this counterfactual data yields a comparatively small student model that is 6% (absolute) more robust and generalizes 5% better across distributions than baselines on various challenging evaluations. This model is also 15% more sensitive in differentiating original and counterfactual examples, on three evaluation sets written by human workers and via human-AI collaboration.
translated by 谷歌翻译
Multi-document summarization (MDS) has traditionally been studied assuming a set of ground-truth topic-related input documents is provided. In practice, the input document set is unlikely to be available a priori and would need to be retrieved based on an information need, a setting we call open-domain MDS. We experiment with current state-of-the-art retrieval and summarization models on several popular MDS datasets extended to the open-domain setting. We find that existing summarizers suffer large reductions in performance when applied as-is to this more realistic task, though training summarizers with retrieved inputs can reduce their sensitivity retrieval errors. To further probe these findings, we conduct perturbation experiments on summarizer inputs to study the impact of different types of document retrieval errors. Based on our results, we provide practical guidelines to help facilitate a shift to open-domain MDS. We release our code and experimental results alongside all data or model artifacts created during our investigation.
translated by 谷歌翻译
In data-driven systems, data exploration is imperative for making real-time decisions. However, big data is stored in massive databases that are difficult to retrieve. Approximate Query Processing (AQP) is a technique for providing approximate answers to aggregate queries based on a summary of the data (synopsis) that closely replicates the behavior of the actual data, which can be useful where an approximate answer to the queries would be acceptable in a fraction of the real execution time. In this paper, we discuss the use of Generative Adversarial Networks (GANs) for generating tabular data that can be employed in AQP for synopsis construction. We first discuss the challenges associated with constructing synopses in relational databases and then introduce solutions to those challenges. Following that, we organized statistical metrics to evaluate the quality of the generated synopses. We conclude that tabular data complexity makes it difficult for algorithms to understand relational database semantics during training, and improved versions of tabular GANs are capable of constructing synopses to revolutionize data-driven decision-making systems.
translated by 谷歌翻译